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ABSTRACT 


This paper presents a study which aims to provide an understanding of the impact of using the components of a digital 
mobile-based behavior change intervention (mBCI) to support critical thinking skills during university student research 
projects. The digital behavior change interventions are tools and techniques designed to induce behavior change and provide 
continuous support and tailored advice for willing learners through web and mobile platforms. We investigated the impact 
and the usability of the designed tools by analyzing self-reflections and the users on the digital mobile-based behavior 
change intervention components. An instrument was used to examine the differences in the self-perceived improvement of 
critical thinking between the intervention group and the control group before and after working on real research projects 
for two months. The results of comparing post surveys for the independent samples showed that the intervention group had 
a statistically significant perceived improvement in critical thinking than the control group. The findings indicated 
encouraging and positive feedback on the use of mobile intervention components to promote critical thinking when 
supervising research projects. 
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1. INTRODUCTION 


Critical thinking is an essential tool for students to be able to reasonably analyze, understand and evaluate 
arguments in their research projects (Carpi et al., 2017; Yilmaz and Keser, 2016). Research projects are 
research-based tasks that enable learners to study a specific problem by examining the possible solutions to be 
supported with reliable evidence (Devi et al., 2017). Students are usually supervised throughout their research 
projects. According to Ismail et al. (2017) and Clear et al. (2014), traditional face-to-face meetings are 
commonly used to facilitate communication between students and their supervisors. Technology has been used 
to partly overcome the communication barriers of time and place (Seifi et al., 2014). Critical thinking, however, 
is important to be individually supported in research projects and individually tailored to students’ levels of 
critical thinking (Brew and Mantai 2017; Duron et al., 2006). Delivering scalable critical thinking content with 
relevant advice to students during their research projects through technical tools must be compatible with the 
nature of critical thinking (Haghparast et al., 2014). It is necessary to design appropriate tools to assist both 
students and supervisors to enrich the individual learning experience when critical thinking is considered. 
A few research gaps have been identified in using technology in critical thinking, and these deficiencies could 
be the cause of the lack of critical thinking. First, there are inconsistencies in definitions to comprehensively 
cover critical thinking, not only as a skill but also as a lifelong behavior that requires sufficient time to 
accumulate (Al-Mubaid and Bettayeb, 2017; Paul and Elder, 2013). Second, there has also been little research 
in dealing with critical thinking and research projects together as relatively associated tasks (Brew and Mantai 
2017; Yilmaz and Keser, 2016). Finally, there is a lack of well-tested suitable tools to help academic 
supervisors, who may lack the technical knowledge to create mobile or web-based interventions to evaluate 
students’ progress and behavior in critical thinking. In fact, the available tools provide mostly manual settings; 
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once they are developed, they are relatively fixed and inflexible, unable for large-scale reuse or iteratively 
collect randomized data (Swart, 2017; Seifi et al., 2014). 

We suggest that using digital mobile behavioral change intervention (mBCI) methods could provide a 
convenient technical tool to assist students to develop critical thinking and research skills. As argued by Wai 
et al. (2018) and Saade (2012), associating critical thinking with behavior provides an insight into the potential 
success of linking mobile devices to learners with fewer restrictions in time and place. Smart-phones that 
provide interventions could help students improve their critical thinking behaviors during their research work 
(Asiri et al., 2017; Wilde and Zaluska, 2016). This research will study the impact of using a mobile application, 
subdivided down to its components, to encourage behavioral changes to improve critical thinking for 
third-year students in their research projects before and after the intervention. The identified mobile 
components used in this study are: 1) activities and training for leaners to understand critical thinking, 2) tasks 
for practicing critical thinking, 3) a basic information page about the research project of the learner, 4) short 
questionnaires to examine the progress of critical thinking, 5) goals and plans settings, 6) notifications for 
nudges and triggers to keep learners engaged in the mobile intervention, 7) inquiries by the learners, and lastly 
8) providing answers and feedback by the researcher to support critical thinking. The components were 
identified based on a literature review and previous interviews with supervisors and academics; published 
recently by the same authors of this paper. The existing LifeGuide Toolbox software, co-designed by the 
authors, has the potential to be successful in designing mobile intervention components to promote critical 
thinking. The LifeGuide Toolbox software package consists of an authoring tool that enables intervention 
builders with minimal programming background to easily create mobile interventions with the mentioned 
components. As a result, the tool could be widely used by many researchers to provide them with the data that 
they need about students’ behaviors regarding their critical thinking in research projects. The suggestions 
provided by the components of the mobile intervention, provide students with relevant information and regular 
advice to improve their behaviors toward their critical thinking during their research projects. 


2. BACKGROUND 


Within the context of mobile learning, smartphones are suggested to offer superior online learning content 
because of their unique personalization features, flexibility, connectedness, and portability (Wai et al., 2018). 
Mobile technology is one of the most important innovations affecting almost everything in our lives, from 
personal interaction and social communication to education and work. Critical thinking, from this perspective, 
is also an essential daily activity that could be encouraged by mobile technology (Asiri et al., 2017). Attaching 
critical thinking to the daily use of mobile devices could help students monitor decision-making, change 
behaviors, acquire intervention support, and track thinking patterns through their diaries, reminders and goals 
at any time and from anywhere. Several attempts have been made to use mobile technology to promote learning 
and thinking skills in different educational settings. For instance, Alnuaim et al. (2014) examined the impact 
of a contextual mobile learning application to improve students’ design thinking through collaboration between 
learners. Similarly, Wong (2013) designed and used a mobile critique platform that enables and motivates 
designers to co-design with each other and criticize each other’s work. However, generally the results of using 
mobile technology to improve critical thinking for students indicated that students still have low levels of 
critical thinking ability, as addressed by Yilmaz and Keser (2016). Moreover, mobile technology has not been 
used to promote critical thinking in the context of research projects by enabling supervisors with less 
programming background to use tested flexible tools to create and deliver reusable digital mobile-based 
interventions to their students. These deficiencies were resolved in the current study. 


2.1 Digital Behavior Change Interventions (DBCIs) 


Digital behavior change interventions are techniques used to provide continuous tailored advice for learners 
wishing to improve their behavior through web and mobile platforms (Yardley et al. 2016). Digital intervention 
has been used in health education fields to facilitate communication between doctors and patients. Data 
generated from the web-based or mobile-based tools can be studied by evaluating how both experts and users 
interact with the system, when and how the information is delivered to the users, and by identifying behaviors 
patterns (Hargood et al. 2014). As mentioned earlier, the features in the LifeGuide Toolbox software are 
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suitable for study because they flexibly provide cross-platform applications for critical thinking cases. The 
software contains several components that can be broken down for this study such as designing activities, tasks, 
short quizzes, planner, goal setting, sending notifications, and receiving questions from intervention users. 
According to Lustria et al. (2013), a digital behavior change intervention involves three main interacting 
participants, as illustrated in Fig. 1. First, the intervention developers 
(i.e. programmers) provide the necessary tools for digital interactions between intervention experts and users 
through web or mobile platforms. Second, the intervention experts (i.e. researchers) who may lack sufficient 
programming skills but may still design digital interventions for their users. Third, the intervention users who 
are the targets of intervention from the experts, may then pass data back to the experts by using the 
mobile-based or web-based intervention that then permits an iterative improvement of the intervention. 
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Figure 1. Main Actors in Digital Behavior Change Interventions (DBCIs) 


The use of DBCI systems in blended educational contexts can influence higher education in a variety of 
ways. Firstly, for students, the systems can influence their approaches to learning and may stimulate 
communication with their educators or with each other. Secondly, for educators, DBCIs may assist the 
development and selection of online resources and change traditional teaching practices. Thirdly, for 
institutions and researchers, DBCIs can provide large data sets that can be analyzed to investigate more deeply 
the processes of learning and learner behavior. 


3. METHODOLOGY 


The study aims to examine the impact of using a mobile application-based behavior change intervention on 
students’ critical thinking in the context of research projects. A mixed method approach was used to collect 
qualitative and quantitative data from participants. The participants were third-year undergraduate students 
from the Electronics and Computer Science (ECS) program at the university, who were invited to participate 
by using the university’s 3rd year ECS email list. Sixty students who agreed to participate were divided 
randomly into two equal groups; a control group and the group that used the mobile application-based 
intervention for two months. Differences in critical thinking skills before and after the experiment between the 
two groups were explored. 


3.1 Procedure 


As shown in (Table 1), a validated instrument was used as a pre- and post-survey tool, which was designed by 
the authors of this paper, to measure critical thinking in research projects in the context of digital behavior 
change intervention. The instrument was inspired by the Paul-Elder critical thinking framework, which consists 
of two sections; the Intellectual Standards’ statements, and the statements for the Elements of Thought 
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(Paul and Elder, 2013). The participants were asked to self-reflect on their critical thinking skills using the 
instrument based on those statements from the Likert scale: No = 1, Sometimes = 2, Not sure = 3, Usually = 4, 
and Always = 5 using the online iSurvey website forms before and after the experiment. Text boxes were left 
empty for adding explanations or comments by participants at the end of the pre- and post-online surveys. 


Table 1. An Instrument to Assess Critical Thinking in Research Projects for DBCI Context 


Intellectual Standards Elements of Thought 
Clarity: When I write reports or essays, I express my | Purpose: I think purposefully when I set my research 
thinking clearly in different ways and with multiple | objectives by trying to answer what the main goal of my 
supporting examples. work is and why it is important. 
Accuracy: | support my arguments by making sure that | Questions: I use my research questions as a guidance for 
all information is correct and free from errors based on | my thinking to figure out how to solve the research 
the reliable resources. problems. 
Precision: In writing, my words and data used are | Information: The information I use is correct, accurate and 
specifically exact and no more details could be added to | relevant to the purpose and to the questions or issues I am 
explain what I mean. addressing. 
Significance: I essentially focus in my research work on | Inferences: The inferences and conclusions I make 
the most important ideas and crucial facts that would help | logically follow from the evidence with no more or less than 
to make a meaningful point. what is implied in the situation. 
Relevance: In the literature review, everything included | Concepts: I justifiably use concepts, ideas, theories, laws, 
is important, that each part makes a difference and | principles, or hypotheses in thinking to make sense of things 
accordingly I connect my arguments to any reliable | in my research work. 
relevant information. Assumptions: In assumptions, which are the beliefs I take 
Depth: My arguments are thorough by tending to explore | for granted subconsciously or unconsciously, I make sure 
the complexities in the research questions which are | that they are justified by sound evidence. 
addressed profoundly in my answers. Point of view: In my research work, I understand the 
Breadth: I consider additional perspectives and different | limitations of my point of view and I fully consider other 
viewpoints when I think or write in my research work to | relevant reasonable viewpoints. 
look at the problem from various ways. Implications: 1 am aware that the implications of my claims 
Logic: My arguments are reasonable that the thinking is | logically follow from other claims or truths, where 
consistent, and the conclusions follow from the evidence | implications follow from thoughts and consequences follow 
where things make sense step-by-step. from actions. 
Fairness: My arguments are balanced, objective and free 
from hidden biases by considering both positive and 
negative outcomes. 


Once students completed the pre-online survey, another email was sent to invite the intervention group to 
continue participating in the mobile intervention experiment for two months by downloading a mobile 
application from the AppStore or Google Play based on their phone’s operating systems. The mobile 
application-based behavior change intervention (Figure 2) was created by the LifeGuide Toolbox software for 
this experiment. The components of the mobile application are designed to enable users to interact with the 
content of the intervention. The intervention components used in this experiment are broken down into 
1) activities and training for users to understand critical thinking, 2) tasks for users to practice critical thinking, 
3) a basic information page about the research project of the users, 4) short questionnaires to examine the 
progress of critical thinking, 5) goals and plans settings, 6) notifications for nudges and triggers to keep users 
engaged in the mobile intervention, 7) inquiries by the users, and lastly 8) providing answers and feedback by 
the researcher to support critical thinking. The content of the critical thinking activities and tasks were formed 
from two recent books (Cottrell, 2017; Paul and Elder, 2013). During the experiment, participants in the 
intervention group received mobile notifications from the researcher. The types of notification vary based on 
the intervention needed to be supported. The intervention components were mapped into the Intellectual 
Standards and the Elements of Thought to understand which components contributed to enhance certain skills 
in critical thinking. Collecting data about approximately how much time spent in each component, along with 
the texts with responses to those components was a way of assessing the impact of the mobile intervention in 
this research. After finishing the two-month experiment, participants from both groups filled out the 
post-online survey to examine the differences between the two groups in critical thinking. 
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Figure 2. Home Page, Activity Page and Short Quizzes Page in The Mobile Application-based Intervention 


3.2 Data Analysis 


Two main factors in the analysis were used to examine the impact of the components of the digital behavior 
change intervention on participants’ critical thinking and research skills. First, analyzing the data from the pre, 
post and SUS (Usability of the System) surveys provides evidence for perceived improvements, if any, in the 
critical thinking Standards and the Elements of thought for both groups and to examine usability. The SUS test 
consists of ten questions with five response options from strongly agree to strongly disagree, which provides a 
reliable tool for measuring the system usability (Brooke, 1996). Second, analyzing the data from log files for 
the mobile application was used to study the correlation between critical thinking improvement and time spent 
in the mobile application, usage patterns and the level of engagement affected by the mobile notification. 


4. RESULTS AND FINDINGS 


Sixty participants completed both the before and after-online surveys from both intervention and control 
groups; thirty participants for each group. The results of independent t-tests for comparing between mean 
values between the two groups in the pre-surveys showed that there were no statistically significant differences 
(with p < 0.05) in the Intellectual Standards or the Elements of Thought before the experiment was carried out, 
with participants from both groups having equivalent perceived levels of critical thinking skills in research 
projects. However, a paired t-test comparing the post-experimental surveys showed that both groups had 
significantly improved their critical thinking skills, indicating an increase in some Standards and some 
Elements regardless of intervention. Table.1 shows the results of a paired t-test that compares mean values for 
the pre- and post-experimental surveys within the same participants for the control group. There were 
Statistically significant differences after the experiment (with p < 0.05) in the Intellectual Standards: Relevance 
and Logic and in the Elements of Thought: Questions and Information. Similarly, for the intervention group, 
as shown in Table 2, the paired t-test showed that there were statistically significant differences after the 
experiment in the Intellectual Standards: Clarity, Relevance, Breadth, and Logic and in the Elements of 
Thought: Questions, Information and Point of View. 
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Table 2. Paired Samples Test for Pre and Post Surveys: Control Group 


Intellectual Paired Differences t-test df Sig. 
Standards and Mean Std. Std. 95% Confidence (2-tailed) 
Elements of Deviation — Error Interval of the 
Thought Mean Difference 

Lower Upper 
Clarity -.266 .980 178 -.632 .099 -1.490 29 147 
Accuracy -.066 .739 135 -.342 .209 -.494 29 .625 
Precision .200 .610 111 -.027 427 1.795 29 .083 
Relevance -.700 1.441 .263 -1.238 -.161 -2.659 29 013 
Significance -.400 1.404 256 -.924 124 -1.560 29 .130 
Depth .066 .739 135 -.209 342 494 29 .625 
Breadth -.133 1.455 265 -.676 .410 -.502 29 .620 
Logic -.466 1.195 218 -.913 -.020 -2.138 29 041 
Fairness -.333 .994 181 -.704 .037 -1.836 29 .O77 
Purpose -.033 .764 139 -.318 252 -.239 29 .813 
Questions -.433 1.135 .207 -.857 -.009 -2.091 29 .045 
Information -.433 1.040 189 -.821 -.044 -2.282 29 .030 
Inferences -.266 1.048 191 -.658 124 -1.393 29 174 
Concepts -.066 1.172 214 -.504 371 -.311 29 .758 
Assumptions -.300 1.489 271 -.856 256 -1.104 29 279 
Point of view .000 .787 143 -.294 294 .000 29 1.00 
Implications -.100 994 181 -.471 271 -.551 29 586 


Table 3. Paired Samples Test for Pre and Post Surveys: Intervention Group 


Intellectual Paired Differences t-test df Sig. (2- 
Standards and = Mean Std. Std. 95% Confidence tailed) 
Elements of Deviation Error Interval of the 
Thought Mean Difference 

Lower Upper 
Clarity -.800 1.517 277 -1.366 -.233 -2.887 29 .007 
Accuracy -.233 1.040 189 -.621 155 -1.229 29 229 
Precision .200 1.730 315 -.446 .846 .633 29 532 
Relevance -.733 1.638 .299 -1.345 -.121 -2.451 29 021 
Significance -.466 1.525 278 -1.036 102 -1.676 29 105 
Depth -.266 1.529 279 -.837 304 -.955 29 348 
Breadth -.300 .702 128 -.562 -.037 -2.340 29 .026 
Logic -.800 1.186 216 -1.242 -.357 -3.694 29 .001 
Fairness -.200 1.471 268 -.749 349 -.744 29 .463 
Purpose -.166 949 173 -.521 188 -.961 29 344 
Questions -.600 1.191 217 -1.045 -.154 -2.757 29 .010 
Information -.533 1.357 247 -1.040 -.026 -2.151 29 .040 
Inferences -.366 1.629 297 -.974 241 -1.233 29 228 
Concepts .066 1.779 324 -.597 731 .205 29 839 
Assumptions .100 1.539 281 -.474 .674 356 29 725 
Point of view -.666 1.124 .205 -1.086 -.246 -3.247 29 .003 
Implications .066 1.080 197 -.336 .470 338 29 .738 


It was also necessary to use the independent t-test to identify the significance of the perceived improvement 
in participants’ critical thinking skills between the two groups after the experiment (i.e. comparing only the 
post surveys for both groups). The results of independent t-tests for comparing mean values of the post surveys 
between the two groups showed that there were statistically significant differences (with p < 0.05) for the 
intervention group only in the Intellectual Standards: Clarity and in the Elements of Thought: Point of View, 
after the experiment was carried out. Thus, participants in the intervention groups had self-reported that they 
had improved in one Intellectual Standard and in one of the Elements of Thought. The data from the two-month 
mobile intervention from the thirty students in the intervention group were also analyzed. In general, there was 
a modest estimated correlation (with 0.65) between the total time spent on the mobile application and the total 
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scores of the post-survey for each participant. However, there no correlation was found between time spent in 
a certain intervention component and its mapped Standards or Elements. As illustrated in Fig. 3, participants 
spent more time reading the content than typing in the mobile application. The results for the engagement with 
the mobile intervention showed that the amount of time spent in the mobile intervention by all participants 
varied depending on the specific intervention components: project information (66 minutes), first activity 
(136 minutes), second activity (96 minutes), third activity (115 minutes), first task (82 minutes), second task 
(97 minutes), third task (69 minutes), short quizzes (45 minutes), goals and plans (19 minutes), inquiries and 
answers (49 minutes), and feedback (92 minutes). The weekly notifications (feedback and reminders) by 
regularly nudging participants to participate helped to keep them engaged with the mobile application to 
practice critical thinking in research projects. 
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Figure 3. Engagement with the Mobile Intervention Components 


Lastly, the usability of the mobile application as measured by the total mean of SUS scores for the current 
research was 77.3, is above the average score of 75.24 (Bangor et al., 2008) of usability in the area of mobile 
and web interfaces. This indicated that the mobile application-based intervention is useful and easy to use based 
on the calculated overall average of SUS scores. 


5. DISCUSSION, CONCLUSIONS AND FUTURE WORK 


In this study, the mobile behavior change intervention helped show perceived improvement in some critical 
thinking Standards and Elements for the participants who used the mobile components in the context of their 
research projects. The mobile intervention has effectively drawn students’ attention into critical thinking while 
working on their research projects. However, the experiment took place during only the first two months of the 
students’ research projects. This might be the reason for the improvement in some but not all Standards and 
Elements of critical thinking. The overall results indicated that students still need support in their critical 
thinking when working in research projects as reported in the literature as well by Al-Mubaid and Bettayeb 
(2017) and Asiri et al. (2017). As a limitation, the researcher was a replacement for the supervisors to support 
students’ critical thinking with the feedback and answers for the students inquires during the experiment. This 
occurred because of the limited time and busy schedules for the academics in supervision during the semester. 
This may have introduced inaccuracies in outcomes. In future work, further assessment by experts and 
academics will be carried on for the qualitative data gathered from this experiment. More evaluation of the 
mobile log files to understand the effect of the mobile intervention components on the improvement in certain 
Standards and Elements of the critical thinking will be undertaken. To conclude, engaging with these mobile 
components helped students understand, practice, reflect and use critical thinking skills in their research 
projects. The mobile application with its components was deemed by participants to be practical, useful and 
easy to use. 
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